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Abstract 

In this paper we present a backtracking version of the survey propagation algorithm. We 
show that the introduction of the simplest form of backtracking greatly improves the ability 
of the original survey propagation algorithm in solving difficult random problems near the 
sat-unsat transition. 

1 Introduction 

Recently a new heuristic algorithm for solving constrain-satisfaction problems has been proposed 
^ 12]. The algorithm is based on survey propagation and it has been successfully applied to 
the K-sat problem ^ |2] and to the coloring problem j3| • The algorithm is well performing in 
the random instance case, where the factor graph |1] associated to an instance of the problem is 
locally cycle- less (i.e. it has the structure of a tree). 

The properties of this survey propagation algorithm have been studied in details in the case 
of the random 3-sat problem (5j in the limit of very large number ( A^) of literals [Tj El E] • Here 
the relevant parameter is 

« = ^ , (1) 

where M is the total number of clauses. 

The — 1 law [31 IIUI lllj should be valid: for a < Uc all random systems (with probability one 
when N goes to infinity) are satisfiable. On the other hand, for a > Oc no large random system 
(i.e. with probability one when N oo) is satisfiable. An heuristic analysis H] suggests that 
Oc = "* ~ 4.267 where a* can be analytically computed using the survey-propagation equations 
defined later. 

Numerical experiments [S] shows that for very large A, e.g. N = 10^, the survey propagation 
algorithm works up to as ~ 4.252, that is definite less, but very near to ac with a time that is 
approximatively linear in N. In other words in the region as < a < ac the problem is satisfiable 
for large N, but we are unable to find a solution to the problem using the survey decimation 
algorithm. 

The aim of this note is to present a backtracking version of the survey propagation algorithm 
that is somewhat slower, but it able to solve problems where the original survey propagation 
algorithm does not work. In section II we present some general considerations on the backtracking 
strategies in the context of survey propagation. In section III we recall the basic definitions and 
results of the survey propagation approach for random instances. Finally in the last section we 
present the details of a simple implementation of backtracking. 
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2 The basic approach 



The basic idea behind survey propagation is rather simple. In a satisfiability problem there are L 
boolean variable (or literals) a{i) {i = 1,L) that may be true or false. We have to assign a truth 
value to these literals in such a way that a given formula (i.e. the sat-formula) is satisfied. 

If we assign a value to one literal (e.g. the one indexed by k) we remain with a problem with 
L — 1 literals. If we make all the correct assignments, we start from a problem with A'^ literals 
and recursively we reduce it to a null problem. Of course making the correct assignments is a 
non-trivial task. An heuristic approach is the following. 

Let us consider a given problem with L variables and let us call is the total number of 
different truth value assignments that make the sat formula satisfied. In a similar vein we could 
define N{i)T and N{i) f the total number of different truth value assignment of the remaining 
L — 1 literals that make the sat-formula satisfied after that if we assign respectively a true value 
or false value to the i^^ literal. We can thus define 

Pt{^) = , Pf{i) = . (2) 

It may be useful to define also 

P(^) = max(PT(^),PF(^)) • (3) 

Let us suppose to we know the quantities -Pr(^) and -Pf(^)- If we recursively decrease the size 
of the problem from iV to by assigning at each step the literal having the maximal value of 
P{i), we are going to find a correct assignment, if it exist. While it is out of question to compute 
exactly these two functions, it is possible to get some reasonable estimates of them. The survey 
equations (that will be described in the next section) provides an heuristic and efficient way to 
estimate these two functions. 

Generally speaking, giving an heuristic for estimating the functions Pt(«) and Ppii) it is 
possible to use this information to recursively decrease the size of the systems. This is the essence 
of the survey propagation method. Indeed in many cases the survey propagation equations tell us 
an estimate of the probabilities Pr(0 and Ppii) that is the best that can be obtained by a local 
analysis (i.e. an analysis such that for given i Prii) can be computed with a fixed error in a time 
that is of order 1). 

Unfortunately a local analysis is not exact and the survey estimations obtained in this way 
may be not fully correct: it is possible that choices done at the initial stage turn out to be wrong. 
This may be checked by computing the increase in the number of configurations that satisfy the 
sat formula if at the step L we leave free the k*^ literal that was blocked at the step M. This 
quantity will be denoted by 1/I{k). If the literal k has been frozen at the previous step (i.e. 
M = L — 1), we have that 

Pik) = I{k). (4) 

If after freezing the other literals, when the difference L — M becomes large, I{k) decreases, this 
is a signal that a wrong choice has been done at the stepM and we would like to backtrack this 
choice. An algorithm, like survey propagation, that is able to compute the quantities P{k) is 
usually able to compute also the quantities I{k). 

These observations suggest a more general algorithm than pure decimation. In our problem 
we may have N — L literals frozen (i.c their value has been assigned) and L literals free. Our aim 
is to start from L = N and to arrive to L = 0, with the highest possible value of Af at the final 
destination. We can consider two kind of moves: the usual decimation moves (L — > L — 1) and 
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the backtracking moves L ^ L + 1. It seems natural in this approach to select for a decimation 
move the variable with largest P{i), where i labels a variable that is not assigned, and to select 
for a backtracking move the variable with smallest I{k). When 



a combined move, where we assign a value to the variable at im (i-e. P{iM) = Pm) and we 
deassign a value to the variable at km {I (km) = Im), leads to a new configuration with the same 
value of L but with an higher value of TV (at least in the generic case where the points Im and km 
are far away on the factor graph and the corresponding literals are not correlated). 

3 A survey of the survey propagation equations 

The survey propagation equations are described in details in the literature ^EIEIE]- Here we 
present only the basic ideas. 

For a given problem it is convenient to consider the set of all the solutions of the problem, i.e. 
of all the assignments that satisfy the sat formula. We are we are interested not only to know the 
number of all the solutions, but also to know other properties of this set. 

At this end it is convenient to say that two solutions are adjacent if their Hamming distance 
is less than eA^, where e is an appropriate small number. 

It has been suggested that in the limit of large N: 

1. In the interval a < (a^ ~ 3.86 for the 3-sat fS]) the set of all solutions is connected, i.e. 
there is a path of mutually adjacent solutions that joins any two solutions of the set. 

2. In the interval < a < Oc {occ ~ 3.92 for the 3-sat [J) the set of all the solutions breaks in 
an large number of different disconnected regions that are called with many different names 
in the literature [13 UHl HZl US UHl HOI E] (states, valleys, clusters, lumps. . . ). Roughly 
speaking the set of all the solutions can be naturally decomposed into clusters of proximate 
solutions, while solutions belonging to different clusters are not close. 

3. Only in the interval Of, < a < Oc (ab 3.92 for the 3-sat; in other models ai, and may 
coincide) there are literals a that take the same value in all the legal configurations of a 
region. We say that these literals form the backbone of a region. It is important to realize 
that a given literal may simultaneously belong to the backbone of one region and not belong 
to the backbone of an other region. 

4. Only the interval am < a < Oc (om ~ 4.153 for the 3-sat model) the simple survey approach 
is correct |22| . For < a < am the simple survey approach is likely to be an approximation 
and a more refined analysis is needed in order to get exact results. 

The last region is the hardest from the computational point of view. Our analysis is done in 
this region and we will concentrate our efforts on the properties of the backbone. Only in the 
third region the survey propagation equations have a non-trivial solution, i.e. with surveys that 
do not have sj{i) equal to one for all i. The appearance of a non-trivial solution of the survey 
equation marks the entrance in the third region. 

The main quantity we will study is B, i.e. the number of regions with different backbones. As 
far as for large N there is an exponential large number of configuration in each region, B is much 
smaller than M. One usually introduce the entropy j2J S and the complexity ^H] ^ defined as 




(5) 



M = exp(5) 



B = exp(S) . 



(6) 
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In the rest of this note we wih consider B and not TV for two reason: 

• It is much simpler to compute B that J\f. 

• It can be argued that the quantity B is more interesting than TV because in the limit a — > a~ , 
S is a quantity of order 1, while TV is exponentially large {nolo sumere, nondum acerbum 
est). In other words the entropy density {S/N) does not vanish in the limit, while the 
complexity density {T,/N) vanishes. 

In a similar way the functions PT{i) and Ppii) will be defined as the ratio of the appropriate 
quantities B (we must substitute TV with B in eq. ((2j). One hopes that this does not make a great 
difference. 

In the survey approach our aim is to compute B and the surveys s{i) where s is a three 
component vector s = {st, sj, sp} (the sum of the components is one). The quantity sr(«) is 
the fraction of backbones that contain the literal at i with value true, sp^i) is the fraction of 
backbones that contains the literal at i with value false si{i) is the fraction of backbones that do 
not contain the literal at i. If for any i sj{i) = 1 backbones are not present and the problem is 
easy. 

From the point of view or our paper the surveys are extremely interesting because with very 
good approximation we have that 

Prii) = 1 - SF{i) , PfH) = 1 - srii) (7) 

More precisely it can be shown that -Pt(0 = 1 — spii) + 0{{1 — SF{i)Y) that is good enough for 
us because in most of the cases we are interested to those values of i such that Pt (or Pp) is very 
near to 1. 

Unfortunately it is not possible to write local equations directly for the surveys, but we have 
to introduce the cavity surveys [23 El El E], i-e. the value that the survey at i would have if 
we remove from the problem the clause c (under the conditions that the literal at i enters in the 
clause c). On large random problem the cavity surveys s{i,c) satisfy simple equations that are 
called the survey propagation equations [T]. 

The solution of these equations may be used to compute both B and the surveys s{i). 

4 The survey algorithm 
4.1 Survey decimation algorithm 

The survey decimation algorithm has been proposed ElEl ClIHj for finding the solution of the 
random K-satisfiability problem j^l El • 

We start by solving the survey propagation equation and to estimate the function P{i). The 
main step in the decimation procedure consists in starting from a problem with L variables and 
in arriving to a problem with L — 1 variables where s{i) is fixed to be true (or false), i.e. the 
literal at i has an assigned value. We have already remarked that if P{i) is near to 1, the second 
problem it is easier to solve: it has nearly the same number of solutions of the belief equations 
and one variable less. (We assume that the complexity can be reliable computed by solving the 
survey propagation equations). 

The decimation algorithm proceeds as follows. We reduces by one the number of variables 
choosing the node i in the appropriate way, e.g. by select the literal with maximal P{i). In a 
slightly less efficient but faster version we select R literals with maximal P, where R = fN and 
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/ is a small number. If we are not too near the critical point and / < .01 the results are only 
weakly dependent on /. 

We recompute the solutions of the survey equations and we reduce again the number of vari- 
ables; it is convenient to monitor the complexity. At the end of the day different things may 
happen: 

• We arrive to a negative complexity (in this case the reduced problem should have no solutions 
and we are lost). 

• If in spite of this warning we go on with the algorithm, we usually find that the reduced 
problem contains a contradiction and in this case we are definitively lost. Sometimes we can 
find a contradiction also for positive complexity, but this happens always in the region of 
very small complexity density. 

• Other troubles arise when the survey equations do not have anymore an unique solution 
and the usual iterative procedure for finding the solution of the survey equations does not 
converge. It is possible that one could find some way to escape from the troubles, but this 
case has never been studied in details. 

• After a sufficient large number of iterations the survey equation do not have any more a 
non-trivial solution and the standard iteration algorithm converges to the trivial solution 
{sT{i) = spii) = 0, s/(i) = 1). If this happens the reduced problem is easy and it can be 
solved using other algorithms. 

• An other possible happy end is that the survey equations converge to a zero entropy solution 
where (depending on the site) or sr(^) = 1 or SF{i) = 1. In this case we do not need other 
algorithms to solve the reduced problem. 

A careful analysis 8^ of the results for large, but finite = O(IO^), shows that the algorithm 
works in the limit of infinite up to as{f) where in the limit of / ^ we have as ~ 4.252, that 
is definite less, but very near to Oc- 

It is interesting to note that for a < as the survey decimation algorithm takes a time that is 
polynomial in A^ and using a smart implementation the time is nearly linear in A^. The survey 
algorithm is an incomplete search procedure that may not be able to find a solution to a satisfiable 
instance. This actually happens with a non-negligible probability e.g. for sizes of the order of a 
few thousands of variables also when a < as, however in the limit A^ oo, it should work as 
soon a < as- In the interval as < a < ac the decimation procedure leads to a regime of negative 
complexity so that the algorithm does not work. 

Unfortunately at the present moment the value of as can be obtained only analyzing how the 
argument works on a finite sample and we are unable to compute analytically as- it is a rather 
difficult task to understand in details why for as it so near to ac- 

4.2 Backtracking surveys 

In the backtracking version of the survey propagation algorithm we follow the general strategy 
outlined in the introduction. 

In order to solve a sat problem with A^ literals we consider auxiliary problems where L literals 
are free and N — L literals have a value assigned. Obviously for given L we have 2^~^ of such 
problems. 
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Figure 1: The results for the S as function of L for one 3-sat problem with = 10^ and a = 4.25 
for the algorithm with / = 10^^ and three values of r (0, .25 and .4, respectively full line, short 
dashed line and long dashed line). 



Our aim is to start from L = and to arrive to to L = N , with an assignment to the literals 
such that the sat formula is satisfied. An heuristic argument suggests that a reasonable strategy 
is to keep the highest possible value of A/" at each intermediate step. 

We can consider two kind of moves: the usual decimation moves (L — L + 1 ) and the 
backtracking moves {L ^ L — \). In the simplest version of the the algorithm we select for 
applying the decimation rule the literal with largest P{i), where i labels a literal that is not 
assigned, and we select for a backtracking move the literal with smallest I{k). The computation 
of P{i) is standard and I{h) can be just computed in the same way as function of the surveys of 
the nodes adjacent to the k*^ node. 

Once we know how to select the literals to decimate and the literals to backtrack, the most 
important decision is if we do a decimation or a backtracking move. 

The simplest possibility is do decimations moves and backtracking moves at a fixed ratio. This 
algorithm depends on the parameter r, i.e. the ratio between the number of backtracking moves 
and the total number of moves (obviously we must have r < 0.5). This choice has the advantage 
that the total number of moves is still bounded by a number proportional to N (more precisely 
by N/{0.5 — r)). If it works, it is very efficient. 

We have done a few numerical experiments and we have found that the introduction of the 
backtracking moves with the simple strategy outline before definitely increases the range of appli- 
cability of the survey equation algorithm. 

In fig. 1. we show the results for the S as function of L for a 3-sat problem with = 10^ and 
a = 4.25 for the algorithm with / = 10"^ with three values of r (0, .25 and .4). In fig. 2 we plot 
the quantity 

p(^l^ = ^ignon-frozen \^T{i) — sr(^)| 
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Figure 2: The results for the the function F as function of L for the same problem of fig. (1) 
{N = 10^, a = 4.25 and / = 10~^) with three values of r (0, .25 and .4, respectively full line, 
short dashed line and long dashed line). 

where the sum is done only on the L non-frozen nodes. The vanishing of F implies that the 
solution of the survey equation is trivial. 

For r = 0, i.e. the standard survey algorithm, the complexity arrives to zero with F ^ 
and the survey algorithm fails to solve the problem. The further evolution (not shown) gives a 
negative complexity and after some more iterations we find a contradiction and the algorithms 
must stops (as expected). The fact that the survey algorithm fails for a value of a very near, but 
smaller than 05 is due to fluctuations in the value of as that are present for the finite value of 
and vanish when N ^ 00. 

On the contrary both for r = .25 and r = .4 we arrive to a situation where the complexity and 
F simultaneously go to zero, i.e. to the trivial solution of the survey equations. The backtracking 
algorithm performs better that the non-backtracking one. 

The backtracking algorithms is able to find an easy reduced problem also in the case where 
the non-backtracking algorithm fails. It is interesting to note that increasing r, the complexity 
at fixed L increases signaling a more efficient algorithms. Indeed there are other cases (e.g. for 
slightly larger a) where the r = .25 algorithm fails and the r = .4 is successful. 

The same phenomenon happens in a consistent way at different values of / and N: also this 
simple version of the backtracking algorithm is more performing that the non-backtracking one 
for difficult problems. 

More carefully and systematic investigations are needed in order find out the dependence of 
aA_ on the backtracking ratio r and to study how near we can go to the critical point (oc) using 
this method. 

It is rather likely that the backtracking method in its present form does not work too near to Oc- 
Indeed the first decrease of the complexity (for L near N) is weakly sensitive to the backtracking; 
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it is likely using a survey propagation algorithm we must enter in a region of negative complexity 
if we are sufficiently near to Oc- In principle the algorithm is able to recover and to increase the 
complexity when we decrease L: this exactly what happens in a few cases for = O(IO^). 

Unfortunately, when we are too near to Oc and we are deep in the region of negative complexity, 
the iterative procedure for finding the solution of the survey propagation equations does not 
converge anymore and the algorithm stop to work. The non-convergence of the iterative procedure 
for finding the solution of the survey propagation equations is a sign of the existence of multiple 
solutions of the survey propagation equations. It is likely that the next step to do is to implement 
a version of the algorithm that bypass this difficulty. It would also be very interesting to do a 
more careful analysis of different strategies in doing backtracking steps. The relative efficiency of 
backtracking in non-random instances is also a problem that one should investigate carefully. 
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